Combining Objective and Subjective Data in Evaluation of Spoken Dialogues
نویسنده
چکیده
Evaluation of human-computer spoken dialogues is often based on analyses of objective metrics such as task completion rate, turn-taking, time consumption, etc. While these data are easily obtained and processed from log files, they offer or very little information about the actual usability of the given service as perceived by the test subjects. This study combines the evaluation based on objective data obtained from log files with subjective data, where the test subjects express their attitudes to a number issues directly related to the usability of the service. It is shown how the joint analysis can be used to support, but also question findings from either source.
منابع مشابه
Core Units of Spoken Grammar in Global ELT Textbooks
Materials evaluation studies have constantly demonstrated that there is no one fixed procedure for conducting textbook evaluation studies. Instead, the criteria must be selected according to the needs and objectives of the context in which evaluation takes place. The speaking skill as part of the communicative competence has been emphasized as an important objective in language teaching. The pr...
متن کاملSubjective experiments on influence of response timing in spoken dialogues
To verify the validity of analysis results relating to dialogue rhythm from earlier studies, we produced spoken dialogues based on analysis results relating to response timing and the other spoken dialogues, and performed subjective experiments to investigate parameters such as the naturalness of the dialogue, the incongruity of the synthesized speech, and the ease of comprehension of the utter...
متن کاملKnowledge-Combining Methodology for Dialogue Design in Spoken Language Systems
In this paper, we propose a strategy for designing dialogue managers in spoken dialogue systems for a restricted domain. This strategy combines several information sources intuition, observation and simulation, in order to maximize the adaptation within the system capability and the expectation of the user. These sources are combined by an iterative process consisting of five steps, where diffe...
متن کاملAdaptive Information Presentation for Spoken Dialogue Systems: Evaluation with real users
We present evaluation results with human subjects for a novel data-driven approach to Natural Language Generation in spoken dialogue systems. We evaluate a trained Information Presentation (IP) strategy in a deployed tourist-information spoken dialogue system. The IP problem is formulated as statistical decision making under uncertainty using Reinforcement Learning, where both content planning ...
متن کاملTracking the Sequences of Patient-Therapist Dialogues by Coding Responses during Integrative Psychotherapy
Aim: The Assimilation of Problematic Experiences Scale (APES) for the coding of client responses and the Process Focused Conversation Analysis (PFCA) for coding therapist responses were applied to transcripts of a successful case of integrative psychotherapy of depression. Methods: the research method of the present research was case study. Dialogues (150) between a therapist and client in one ...
متن کامل